AutoML Leaderboard

Best model	name	model_type	metric_type	metric_value	train_time
	1_Baseline	Baseline	logloss	0.687821	1.87
	2_DecisionTree	Decision Tree	logloss	0.454124	18.11
the best	3_Default_Xgboost	Xgboost	logloss	0.267911	25.59
	4_Default_NeuralNetwork	Neural Network	logloss	0.485527	15.27
	5_Default_RandomForest	Random Forest	logloss	0.393335	20.94
	Ensemble	Ensemble	logloss	0.267911	1.64

AutoML Performance

AutoML Performance Boxplot

Features Importance

features importance across models

Spearman Correlation of Models

models spearman correlation

Summary of 2_DecisionTree

<< Go back

Decision Tree

n_jobs: -1
criterion: gini
max_depth: 3
explain_level: 2

Validation

validation_type: split
train_ratio: 0.75
shuffle: True
stratify: True

Optimized metric

logloss

Training time

16.6 seconds

Metric details

	score	threshold
logloss	0.454124	nan
auc	0.858947	nan
f1	0.825033	0.310846
accuracy	0.792628	0.310846
precision	0.94211	0.841315
recall	1	0.0420701
mcc	0.581852	0.310846

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.454124	nan
auc	0.858947	nan
f1	0.825033	0.310846
accuracy	0.792628	0.310846
precision	0.771592	0.310846
recall	0.886429	0.310846
mcc	0.581852	0.310846

Confusion matrix (at threshold=0.310846)

	Predicted as long	Predicted as short
Labeled as long	1343	640
Labeled as short	277	2162

Learning curves

Decision Tree

Tree #1

Rules

if (Total Costs <= 8584.205) and (Total Costs <= 4865.995) and (CCS Diagnosis Code <= 253.5) then class: short (proba: 95.64%) | based on 3,278 samples

if (Total Costs <= 8584.205) and (Total Costs > 4865.995) and (CCS Diagnosis Code <= 454.5) then class: short (proba: 72.63%) | based on 2,769 samples

if (Total Costs > 8584.205) and (APR Severity of Illness Code <= 2.5) and (APR Medical Surgical Description > 0.5) then class: short (proba: 57.49%) | based on 1,868 samples

if (Total Costs > 8584.205) and (APR Severity of Illness Code <= 2.5) and (APR Medical Surgical Description <= 0.5) then class: long (proba: 75.53%) | based on 1,847 samples

if (Total Costs > 8584.205) and (APR Severity of Illness Code > 2.5) and (Total Costs > 15573.525) then class: long (proba: 95.33%) | based on 1,797 samples

if (Total Costs > 8584.205) and (APR Severity of Illness Code > 2.5) and (Total Costs <= 15573.525) then class: long (proba: 78.08%) | based on 958 samples

if (Total Costs <= 8584.205) and (Total Costs <= 4865.995) and (CCS Diagnosis Code > 253.5) then class: short (proba: 68.65%) | based on 453 samples

if (Total Costs <= 8584.205) and (Total Costs > 4865.995) and (CCS Diagnosis Code > 454.5) then class: long (proba: 86.44%) | based on 295 samples

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Kolmogorov-Smirnov Statistic

Precision-Recall Curve

Calibration Curve

Cumulative Gains Curve

Lift Curve

SHAP Importance

SHAP Dependence plots

Dependence (Fold 1)

SHAP Dependence from Fold 1

SHAP Decision plots

Top-10 Worst decisions for class 0 (Fold 1)

SHAP worst decisions class 0 from Fold 1

Top-10 Best decisions for class 0 (Fold 1)

SHAP best decisions class 0 from Fold 1

Top-10 Worst decisions for class 1 (Fold 1)

SHAP worst decisions class 1 from Fold 1

Top-10 Best decisions for class 1 (Fold 1)

SHAP best decisions class 1 from Fold 1

<< Go back

Summary of 1_Baseline

<< Go back

Baseline Classifier (Baseline)

n_jobs: -1
explain_level: 2

Validation

validation_type: split
train_ratio: 0.75
shuffle: True
stratify: True

Optimized metric

logloss

Training time

0.8 seconds

Metric details

	score	threshold
logloss	0.687821	nan
auc	0.5	nan
f1	0.710975	0.496442
accuracy	0.55156	0.496442
precision	0.55156	0.496442
recall	1	0.496442
mcc	0	0.496442

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.687821	nan
auc	0.5	nan
f1	0.710975	0.496442
accuracy	0.55156	0.496442
precision	0.55156	0.496442
recall	1	0.496442
mcc	0	0.496442

Confusion matrix (at threshold=0.496442)

	Predicted as long	Predicted as short
Labeled as long	0	1983
Labeled as short	0	2439

Learning curves

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Kolmogorov-Smirnov Statistic

Precision-Recall Curve

Calibration Curve

Cumulative Gains Curve

Lift Curve

<< Go back

Summary of 4_Default_NeuralNetwork

<< Go back

Neural Network

n_jobs: -1
dense_1_size: 32
dense_2_size: 16
learning_rate: 0.05
explain_level: 2

Validation

validation_type: split
train_ratio: 0.75
shuffle: True
stratify: True

Optimized metric

logloss

Training time

14.0 seconds

Metric details

	score	threshold
logloss	0.485527	nan
auc	0.912011	nan
f1	0.857029	0.272604
accuracy	0.839665	0.369153
precision	0.960199	0.975159
recall	1	3.31565e-48
mcc	0.676713	0.369153

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.485527	nan
auc	0.912011	nan
f1	0.8533	0.369153
accuracy	0.839665	0.369153
precision	0.86132	0.369153
recall	0.845428	0.369153
mcc	0.676713	0.369153

Confusion matrix (at threshold=0.369153)

	Predicted as long	Predicted as short
Labeled as long	1651	332
Labeled as short	377	2062

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Kolmogorov-Smirnov Statistic

Precision-Recall Curve

Calibration Curve

Cumulative Gains Curve

Lift Curve

<< Go back

Summary of 3_Default_Xgboost

<< Go back

Extreme Gradient Boosting (Xgboost)

n_jobs: -1
objective: binary:logistic
eta: 0.075
max_depth: 6
min_child_weight: 1
subsample: 1.0
colsample_bytree: 1.0
eval_metric: logloss
explain_level: 2

Validation

validation_type: split
train_ratio: 0.75
shuffle: True
stratify: True

Optimized metric

logloss

Training time

24.0 seconds

Metric details

	score	threshold
logloss	0.267911	nan
auc	0.95594	nan
f1	0.901609	0.500049
accuracy	0.891226	0.520298
precision	0.997674	0.995696
recall	1	0.000343896
mcc	0.780103	0.520298

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.267911	nan
auc	0.95594	nan
f1	0.901414	0.520298
accuracy	0.891226	0.520298
precision	0.90123	0.520298
recall	0.901599	0.520298
mcc	0.780103	0.520298

Confusion matrix (at threshold=0.520298)

	Predicted as long	Predicted as short
Labeled as long	1742	241
Labeled as short	240	2199

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Kolmogorov-Smirnov Statistic

Precision-Recall Curve

Calibration Curve

Cumulative Gains Curve

Lift Curve

SHAP Importance

SHAP Dependence plots

Dependence (Fold 1)

SHAP Dependence from Fold 1

SHAP Decision plots

Top-10 Worst decisions for class 0 (Fold 1)

SHAP worst decisions class 0 from Fold 1

Top-10 Best decisions for class 0 (Fold 1)

SHAP best decisions class 0 from Fold 1

Top-10 Worst decisions for class 1 (Fold 1)

SHAP worst decisions class 1 from Fold 1

Top-10 Best decisions for class 1 (Fold 1)

SHAP best decisions class 1 from Fold 1

<< Go back

Summary of 5_Default_RandomForest

<< Go back

Random Forest

n_jobs: -1
criterion: gini
max_features: 0.9
min_samples_split: 30
max_depth: 4
eval_metric_name: logloss
explain_level: 2

Validation

validation_type: split
train_ratio: 0.75
shuffle: True
stratify: True

Optimized metric

logloss

Training time

19.3 seconds

Metric details

	score	threshold
logloss	0.393335	nan
auc	0.904638	nan
f1	0.851336	0.358692
accuracy	0.825192	0.58604
precision	0.978788	0.964385
recall	1	0.0399946
mcc	0.648041	0.58604

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.393335	nan
auc	0.904638	nan
f1	0.839327	0.58604
accuracy	0.825192	0.58604
precision	0.85118	0.58604
recall	0.827798	0.58604
mcc	0.648041	0.58604

Confusion matrix (at threshold=0.58604)

	Predicted as long	Predicted as short
Labeled as long	1630	353
Labeled as short	420	2019

Learning curves

Permutation-based Importance

Confusion Matrix

Normalized Confusion Matrix

ROC Curve

Kolmogorov-Smirnov Statistic

Precision-Recall Curve

Calibration Curve

Cumulative Gains Curve

Lift Curve

SHAP Importance

SHAP Dependence plots

Dependence (Fold 1)

SHAP Dependence from Fold 1

SHAP Decision plots

Top-10 Worst decisions for class 0 (Fold 1)

SHAP worst decisions class 0 from Fold 1

Top-10 Best decisions for class 0 (Fold 1)

SHAP best decisions class 0 from Fold 1

Top-10 Worst decisions for class 1 (Fold 1)

SHAP worst decisions class 1 from Fold 1

Top-10 Best decisions for class 1 (Fold 1)

SHAP best decisions class 1 from Fold 1

<< Go back

Summary of Ensemble

<< Go back

Ensemble structure

Model	Weight
3_Default_Xgboost	5

Metric details

	score	threshold
logloss	0.267911	nan
auc	0.95594	nan
f1	0.901609	0.500049
accuracy	0.891226	0.520298
precision	0.997674	0.995696
recall	1	0.000343896
mcc	0.780103	0.520298

Metric details with threshold from accuracy metric

	score	threshold
logloss	0.267911	nan
auc	0.95594	nan
f1	0.901414	0.520298
accuracy	0.891226	0.520298
precision	0.90123	0.520298
recall	0.901599	0.520298
mcc	0.780103	0.520298

Confusion matrix (at threshold=0.520298)

	Predicted as long	Predicted as short
Labeled as long	1742	241
Labeled as short	240	2199